A Self-Organizing Network that Can Follow Non-stationary Distributions

A new on-line criterion for identifying \useless" neurons of a self-organizing network is proposed. When this criterion is used in the context of the (formerly developed) growing neural gas model to guide deletions of units, the resulting method is able to closely track nonstationary distributions. Slow changes of the distribution are handled by adaptation of existing units. Rapid changes are handled by removal of \useless" neurons and subsequent insertions of new units in other places. 1 Non-stationary data is di cult to handle : : : Non-stationary data distributions can be found in many technical, biological or economical processes. Self-organizing neural networks have rarely been considered for tracking those distributions since many of the models, e.g. the selforganizing map [6], neural gas [8], or the hypercubical map [1], use decaying adaptation parameters. Once the adaptation strength has decayed, the network is \frozen" and thus unable to react to subsequent changes in the signal distribution. 2 : : : even for incremental networks Models with small constant parameters such as the incremental networks developed by the author [2{4] are in a somewhat better position for handling non-stationary distributions. The non-decreasing adaptation rate enables the networks to follow slowly changing probability distributions like e.g. a normal distribution with a slowly drifting mean. Rapid changes in the distribution, however, can in general not be handled properly as is illustrated for the growing neural gas (GNG) [3] model in gure 1. In the case of GNG it is easy to see why considerably many units may get stuck in former regions of high probability density and become so-called dead units. The network topology is updated { apart from the characteristic insertions { by two mechanisms (as described in detail in [3]). \Competitive Hebbian Learning" [7] is used to create new connections by always inserting a connection between the units s1 and s2 nearest and second-nearest to the current input 1 : : : as does the classical k-means algorithm.